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The Massachusetts Department of Elementary and Secondary Education supports low-performing schools 
through a process that draws on qualitative and quantitative data from monitoring visits. The data are 
used to produce ratings for 26 turnaround indicators in four turnaround practice areas relating to school 
leadership, instructional practices, student supports, and school climate. This study analyzed data on 
school indicator ratings collected during school years 2014/15—2018/19 from 91 low-performing schools, 
with a focus on the distribution of the ratings among schools during their first year in the monitoring 
system and on the relationship of ratings to school outcomes. During the first year in which ratings data 
were available for a school, a majority of schools were in the two highest rating levels for 21 of the 26 
indicators. Schools generally had lower rating levels for indicators in the student supports practice area 
than in the other three practice areas. Ratings for half the indicators were statistically significantly related 
to better schoolwide student outcomes and had a practically meaningful effect size of .25 or greater, and 
none was statistically significantly related to worse outcomes. Two indicators in the leadership practice 
area (school leaders’ high expectations for students and staff and trusting relationships among staff) were 
related to lower chronic absenteeism rates. Ratings for five indicators in the instructional practices area 
were related to higher student academic growth in English language arts or math; two of these indicators 
(use of student assessment data to inform classroom instruction and school structures for instructional 
improvements) were related to higher growth in both English language arts and math. Ratings for four 
indicators in the student supports practice area (teacher training to identify student needs, research- 
based interventions for all students, interventions for English learner students, and interventions for 
students with disabilities) were related to higher student academic growth in English language arts or 
math. Two indicators in the school climate practice area (schoolwide behavior plans and adult—student 
relationships) were related to higher student academic growth in English language arts or math or lower 
chronic absenteeism rate. Eight indicators were not statistically related to any of the outcomes of interest. 


The Massachusetts Department of Elementary and Secondary Education (DESE) has developed state systems 
of support for low-performing schools, followed by a systemic monitoring process that focuses on school turn- 
around practices in four areas: leadership, instruction, student support, and school climate. During the moni- 
toring process, DESE draws on qualitative and quantitative methodologies to rate low-performing schools on a 
rubric, called the Massachusetts Turnaround Practices and Indicators (TP&l), that reflects the four turnaround 
practice areas.' 


1. American Institutes for Research, & Massachusetts Department of Elementary and Secondary Education. (2015). Massachusetts moni- 
toring site visits turnaround practices and indicators continuum. Massachusetts Department of Elementary and Secondary Education. 
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The monitoring system has been in place since the 2014/15 school year, and schools identified as low-performing 
through the state’s school accountability system receive annual monitoring visits and reports that assess school 
progress, provide formative feedback, and inform continuous improvement priorities along with areas for target- 
ed state support. The TP&l rating system enables DESE to examine the progress of low-performing schools in a 
single year and across years. DESE can use the TP&l rating system to identify both common and unique areas of 
need in the state’s low-performing schools and allocate resources and supports accordingly. 


In alignment with research on school improvement and change management,’ DESE is interested in knowing 
whether the ratings for schools entering the system for the first time varied, as might be expected, and how much 
room the ratings leave for improvement. In addition, DESE’s goal is to deepen its understanding of school turn- 
around and to provide schools with focused feedback on turnaround indicators that are strongly related to school 
outcomes. Findings from this study will inform DESE’s continuous improvement efforts and increase its ability to 
focus on indicators with ratings that are related to school outcomes, thereby devoting its limited resources and 
capacity to practices that are more likely to make a difference for school improvement. 


What was studied and how? 


The study addressed the following research questions by examining available data on the TP&lI ratings and key 
school outcomes over 2014/15—-2018/19: 


1. In each turnaround practice area, what is the distribution of the TP&l ratings in Massachusetts low-performing 
schools when they first entered the monitoring system? 


2. In each turnaround practice area, to what extent are the TP&I ratings correlated with two schoolwide student 
outcomes—school mean student growth percentile (in English language arts and math) and school chronic 
absenteeism rate—after school demographic characteristics are controlled for? 


The study drew on five years of monitoring data from 91 schools. The data for the study include TP&l ratings for 
the lowest performing schools in Massachusetts in 2014/15—2018/19, schools that were typically in the bottom 
10 percent on schoolwide annual student academic achievement and academic achievement growth. These data 
are publicly available from DESE. The sample for this study included all 91 schools that received initial monitoring 
visits during 2014/15-2018/19 because of their identification as a low-performing school in the state’s account- 
ability system. Of these 91 schools, 41 were elementary schools, 25 were middle schools, 21 were high schools, 
and 4 were K-8 schools. A total of 229 monitoring visits to these schools were conducted over 2014/15—2018/19. 


The study examined two schoolwide student outcomes: school mean student growth percentile (in English lan- 
guage arts and math) and school chronic absenteeism rate. School mean student growth percentile (SGP) is a 
school-level, aggregated form of SGP that compares a student’s score with the scores of all students in the state 
in the same grade who received similar scores in prior years. In this study the school mean SGP reflects a school’s 
performance in student achievement. 


Monitoring rates of chronic absenteeism is part of the state’s school accountability mechanism. To be considered 
chronically absent, a student must miss 10 percent or more of school days in a school year, the equivalent of 18 
or more days in a 180 day school year. The school chronic absenteeism rate is the percentage of students in the 
school who are chronically absent each year. 


2. Murphy, J. (2009). Turning around failing schools: Policy insights from the corporate, government and nonprofit sectors. Education 
Policy, 23(6), 796-830. 
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For research question 1 the study team examined the percentage distributions of implementation ratings for TP&l 
turnaround indicators for low-performing schools in Massachusetts in the first year in which data were available 
for each school. First-year data provided information about the implementation status of these indicators for 
schools when they received their first monitoring visits. During the first year of monitoring visits, schools received 
TP&l ratings and started to implement practices that are based on the state’s consistent approach to monitoring 
and providing formative feedback to low-performing schools. The analysis was intended to establish a baseline 
of TP&lI ratings and to identify the indicators in the four-point rating system (from limited evidence, O, to sustain- 
ing, 3) that tend to have more room for continuous improvement. 


For research question 2 the study team used two-level hierarchical linear models with years grouped within 
schools to examine the relationship between each turnover indicator and the schoolwide student outcomes, 
while controlling for school demographic characteristics, including grade span, percentages of male and female 
students, percentage of racial/ethnic minority students, percentage of English learner students, percentage of 
students with a disability, and percentage of economically disadvantaged students, and four year dummy vari- 
ables that indicated the nth year for which the school had observation data. The main report describes findings 
from the models that are statistically significant at the p < .05 level, along with the associated effect sizes. 


Findings 


During the first year for which monitoring data were available, at least half of the schools in the sample were 
in the two highest rating levels (providing or sustaining) for 21 of the 26 TP&l indicators. In addition, regression 
analyses for each indicator in the four practice areas revealed significant relationships and effect sizes of .25 
or greater—considered to be practically meaningful in an education setting—between higher indicator ratings 
and higher school mean SGP in English language arts and math and lower chronic absenteeism rate, after school 
demographic characteristics were controlled for (table 1). 


Table 1. Turnaround indicators on the Massachusetts Turnaround Practices and Indicators rubric that 
have a statistically significant and practically meaningful relationship with schoolwide student outcomes, 
2014/15-2018/19 


School mean student growth percentile 


Turnaround practice area and indicator English language arts Math Chronic absenteeism 


1. Leadership, Shared Responsibility, and Professional Collaboration 


11 Use of Autonomy 


1.2 High Expectations and Positive Regard | 


1.3. Vision/Theory of Action and Buy-In 


1.4 Monitoring of Implementation and School Progress 


1.5 Trusting Relationships | 


1.6 Time Use for Professional Development and Collaboration 


1.7 Communication with Staff 


1.8 Sustainability 
2. Intentional Practices for Improving Instruction 


2.1 Instructional Expectations | 


2.2 Instructional Schedule || 


2.3. Identifying and Addressing Student Academic Needs 


2.4 Classroom Observation Data Use 


2.5 Student Assessment Data Use (for schoolwide decision-making) | 
2.6 Student Assessment Data Use (for classroom instruction) | | 
2.7. Structures for Instructional Improvement | | 
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School mean student growth percentile 


Turnaround practice area and indicator English language arts Math Chronic absenteeism 


3. Student-Specific Supports and Instruction to All Students 


3.1 General Academic Interventions and Enrichment | ® 


3.2 Teacher Training to Identify Student Needs (academic and | 
nonacademic) 


3.3. Determining Schoolwide Student Supports 


3.4 Multitiered System of Support (academic and nonacademic) 


3.5 Academic Interventions for English Language Learners 1} 


3.6 Academic Interventions for Students with Disabilities | 
4. School Climate and Culture 
41 Schoolwide Behavior Plan a | 


4.2 Adult-Student Relationships B a 


4.3 Expanded Learning 


4.4 Wraparound Services and External Partners 


4.5 Family and Community Engagement 


Note: Blocks (Ml) indicate a significant relationship between the TP&l indicator and the outcome at the p < .05 level with an effect size of at least .25, 
which is considered a practically meaningful effect. All the significant relationships are in the desired direction (positively related to school mean SGP or 
negatively related to chronic absenteeism rate). Blank cells represent findings that were not statistically significant or that were statistically significant 
but not practically meaningful. Each two-level hierarchical linear model with observations grouped within schools controlled for school demographic 
characteristics. Statistically significant thresholds (p-values) were adjusted in each domain using the Benjamini—Hochberg correction for multiple com- 
parisons method to adjust for multiple comparisons. Analyses were based on 229 year-specific observations from 91 schools. For a full description of 
the rubric of the indicators and the coefficients, see appendix B in the main report. 


Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 


Implications 


DESE plans to use the study findings in aligning the state’s accountability system with its monitoring and support 
systems and to further refine the TP&l rubric. Typically, DESE recommends that schools prioritize a subset of TP&I 
indicators each year based on the student needs and staff strengths at the school. The prevalence of higher ratings 
across the indicators suggests that schools, during their first year of monitoring visits, are focusing on indicators 
of improvement recommended by DESE. The finding that most schools identified as low performing received 
TP&I ratings toward the higher end of the scale might also reflect consensus about which indicators schools chose 
to prioritize. DESE can use the findings to provide feedback on which indicators to prioritize for schools identified 
as low performing. The findings might also suggest that the indicator-specific rating-level descriptions need to be 
adjusted to provide more specificity and distinction between rating levels. 


DESE might want to concentrate its efforts and investments on practice areas or indicators that have a strong 
relationship with school improvements. DESE can use the findings from regression analyses to focus its future 
support for low-performing schools on indicators that are related to improved school outcomes. While the rela- 
tionships are not causal, the findings can help DESE identify a set of indicators that, with further study, might be 
shown to be predictive of school outcomes or that can be used to determine what type of support a school needs. 


Finally, other state education agencies might want to use this study’s approach of aligning the school account- 
ability system with the monitoring process to monitor and support school improvement efforts. This integrative 
approach, with a systemic monitoring process that incorporates school accountability measures, can provide 
valuable formative feedback for continuous improvement of low-performing schools and districts. The findings 
can provide a starting point for other state education agencies that wish to focus on strategies leading to desired 
outcomes that allow schools greater flexibility in implementing and adapting these strategies to support their 
students. 
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